Method and system for processing images acquired in real time through a medical device

ABSTRACT

A method for processing images acquired in real time through a medical device, said images being loaded into a buffer, comprising the steps of: stopping the loading of the images into the buffer, processing loaded images using an incremental algorithm, displaying successively intermediate results of the processing, resuming the loading and stopping the processing based on an evaluation of said intermediate results.

BACKGROUND

1. Field of the Disclosure

The invention relates generally to image and video online processing andin particular to a system and method for processing images acquired inreal time and especially images acquired through a medical device.

2. Background Art

Online processing of the data is critical for applications such as videosurveillance, industrial inspection, robotics and biomedical imaging.For example, video processing may be of interest in endoscopy andendomicroscopy. Patent application US2005207668 presents for example asystem to restore in real-time images acquired through a bundle offiber-optics typically used in endomicroscopy.

Image and video processing techniques are commonly used in digital videoacquisition devices. The main purpose of such algorithms is to extractuseful information from data. This can mean anything from the simplestvisualization enhancement to fully-automatic image-based decision makingduring surgery.

For example, during an endoscopy, the physician's attention might becaught by a specific detail of a video sequence from a given part of atissue. In order to examine the interesting image, the physician mayneed said image to be processed. Online image processing may notably berun through real time processing or lagged-time processing. Real timeprocessing may only be implemented when the processing time is shorterthan the time between two images. Lagged-time processing may only beimplemented when the processing can be completed within a timecorresponding to a fixed number of images and requires to launch inparallel several processes. As lagged processing may lead to loosing thelocation of the investigated area on the tissue, common endoscopysystems provide a freeze function which enables to stop on a givenimage. By freezing upon demand the display, the physician is given moretime to analyze the image and make a diagnosis. Freezing the video atthe exact time asked by the physician may result in freezing a bad,blurred image. U.S. Pat. No. 4,901,143 and U.S. Pat. No. 5,270,810propose a processing that selects a frozen image which is at the sametime a good image and is close to the freezing time asked by theclinician. U.S. Pat. No. 4,901,143 and U.S. Pat. No. 5,270,810 alsodisclose freezing upon demand and address the issue of keeping theinformation contained in the part of the video sequence that occursduring the freeze period by using two parallel pipelines. However,common techniques are essentially limited by the inner quality or amountof information of the frozen images.

The present disclosure proposes a method and a system that enables toenhance information retrieval during ongoing video acquisitions.

SUMMARY OF CLAIMED SUBJECT MATTER

According to one aspect, embodiments described herein relate to a methodfor processing images acquired in real time through a medical device,said images being loaded into a buffer, comprising the steps of:

-   -   stopping the loading of the images into the buffer,    -   processing loaded images using an incremental algorithm,    -   displaying successively intermediate results of the processing,    -   resuming the loading and stopping the processing based on an        evaluation of said intermediate results.

This enables to take advantage of a freeze period for running somecomputationally intensive processing scheme that may not be able to berun in real time. Incremental algorithms are composed of differentsubroutines that need to be run one after the other. The result of eachsubroutine (i.e. an intermediate result) may be of interest in itself.Incremental algorithms may be for example used to find approximatesolutions to problems for which exact solutions cannot be found orcannot be found in a reasonable amount of time such as nondeterministicpolynomial-time hard problems for example. Each intermediate result mayprovide an approximate solution and is thus of interest. The more stepscan be performed, the closer the approximate solution will be to theexact solution as results are improved from one step to the other.Medical devices to acquire images may be any device known to one ofordinary skill in the art including, but not limited to:endomicroscopes, classical endoscopy, High Definition endoscopy, NarrowBand Imaging endoscopy, FICE® endoscopy, double-balloon enteroscopy,zoom endoscopy, 2D/3D ultrasound imaging or any other non irradiativeinterventional modality. The images processed may be consecutive imagesfrom a video sequence or may be a subset of any loaded images.

According to a second aspect, embodiments described therein relate to animaging system comprising:

-   -   a medical device for acquiring images,    -   a storage device comprising a buffer for loading said images,    -   a processor for processing images,    -   a display device,

wherein:

-   -   the processor processes loaded images using an incremental        algorithm after the loading is stopped, intermediate results of        said algorithm are displayed successively by the display device        and the loading is resumed based on an evaluation of said        intermediate results.

The wording “freeze command” refers to stopping the loading into thebuffer. The wording “freeze time” refers to the period of time duringwhich the loading is stopped and the processing may be implemented. Thewording “frozen buffer” refers to the buffer during the freeze time andso on.

Other aspects and advantages of the invention will be apparent from thefollowing description and the appended claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram representing schematically steps of a methodaccording to an embodiment of the present disclosure.

FIG. 2 is a diagram representing schematically steps of a methodimplementing video sequence mosaicing incremental algorithm according toan embodiment of the present disclosure.

FIG. 3 is a display illustrating successive results of a video sequencemosaicing incremental algorithm according to an embodiment of thepresent invention.

FIG. 4 is a diagram representing schematically steps of a methodimplementing a super-resolution incremental algorithm according to anembodiment of the present disclosure.

FIG. 5 is a display illustrating successive results of asuper-resolution incremental algorithm according to an embodiment of thepresent disclosure.

FIG. 6 is a diagram representing schematically steps of a methodimplementing blood velocity measurement incremental algorithm accordingto an embodiment of the present disclosure.

FIG. 7 is a display illustrating successive results of a blood velocitymeasurement incremental algorithm.

FIG. 8 is a diagram representing schematically steps of a methodimplementing an image fusion incremental algorithm according to anembodiment of the present disclosure.

FIG. 9 is a display illustrating successive results of an image fusionincremental algorithm according to an embodiment of the presentdisclosure.

DETAILED DESCRIPTION

The present disclosure relates to an image processing system and methodthat may allow notably computationally intensive video processing, thatcannot run in real-time, to be performed online, upon demand and duringa given amount of time on a frozen set of images taken from a videostream acquired in real time by an acquisition device.

In a basic, not frozen, mode of operation, a video acquisition deviceacts as an input for the system. Real-time video processing may beperformed and the data can be displayed and recorded. In the meantime,the data is queued in a buffer which may be a first in first out (FIFO)finite buffer.

Upon activation of a freeze command, data coming from the videoacquisition device may continue in the potential real-time videoprocessing, recording and display pipeline but may not be queued in theFIFO buffer anymore. Namely, the FIFO buffer is frozen. In the meantime,the computationally intensive algorithm of interest may start working onthe frozen buffer and may continue until the freeze command isdeactivated.

Computationally intensive algorithm are generally incremental algorithmand are processed in several steps, each steps giving intermediateresults. For example, incremental algorithm may be iterative meaningthat after some initialization, an intermediate result is enhanced ateach iteration. Each time such an enhanced processing intermediateresult becomes available, the proposed system may display the result andrecord it.

In general, it is not possible to predict which intermediate result ofthe incremental algorithm might be considered as good enough to stop theprocessing. Therefore, each intermediate result has to be evaluatedbased on at least one of a quantitative criteria, a subjective criteriaand a human criteria in order to know if the processing has to becarried on.

Specific embodiments of the present disclosure will now be described indetail with reference to the accompanying Figures.

FIG. 1 illustrates several steps of a method according to an embodimentof the present disclosure. Once the video acquisition device has started(step 100), the video processing system receives an image stream fromthe video acquisition device (step 110). The image stream may beprocessed in real-time in step 120. Both the original data and thereal-time processed one may be displayed (step 121) and stored (step122). A freeze test is then performed (step 130). If the system is notin frozen mode (arrow N), the images are loaded into a buffer (step140). If the system is turned to freeze mode (arrow Y), new images donot enter the buffer anymore and the computationally intensive algorithmof interest starts processing the set of buffered images (step 150).Computationally intensive algorithms may work in an incremental mannerand provide intermediate results at each completion of a step. At eachcompletion of a step in the algorithm, the system may check whether itis still in freeze mode or not. If the system is still in freeze mode, anew processing step may be launched, otherwise the algorithm is stoppedand the images loading into the buffer is resumed (step 140). In bothcases, intermediate results of the algorithm may also be displayed (step151) and/or stored (step 152).

The display may be done on a motion picture display and the like.Several such devices may be used to display the different video streams.The different streams might also be combined onto a single displaydevice. Simple juxtaposition or advanced image fusion techniques mightbe used.

The storage and the FIFO buffer may be located on a local or remote diskstorage, a memory device and the like. When the system is in the defaultnot-frozen mode, the original or real-time processed images are queuedin a bounded FIFO buffer. If the FIFO buffer is not yet at fullcapacity, the new images are simply appended to the FIFO buffer. If theFIFO buffer is already full, the new images will replace the oldest one.The actual capacity bound of the FIFO buffer may be chosen by the useror by the system or may simply be defined by hardware constraints.

In an embodiment, a user monitors the original or real-time processedimage stream displayed on a display device. When said user sees aninteresting scene and decides that an image processing should be run, hemay for example press a button that may for example be located on theacquisition device, triggering the freeze mode. Going back to thedefault not frozen mode might be triggered for example by releasing thebutton, pushing another button, automatically after a given amount oftime and the like. Freeze mode might also be automatically orsemi-automatically activated or deactivated based on a decision made byanother processing algorithm. Such algorithm may be for example a motiondetection algorithm as disclosed in U.S. Pat. No. 4,901,143 and U.S.Pat. No. 5,270,810. These algorithms may be coupled in order to activatethe freeze mode when a motion on an image stream goes from smooth toerratic.

A computationally intensive algorithm simply aims at extracting usefulinformation from a frozen images set buffer. Thanks to a continuingincrease in the available practical computing power, the complexity ofalgorithms available for image processing tasks has become higher.Advanced processing is now possible in real-time or with some latency.Despite these advances, there will always be a gap between the actualavailable computing power and the computing power required to run someinteresting cutting-edge processing algorithms on the fly. Because ofhardware constraints, extracting an interesting information from a setof images may not always be completed within the time that separates twoframes coming from an acquisition device. In an embodiment somescenarios, being able to run a cutting-edge computationally-intensiveprocessing algorithm during video acquisition may allow the developmentof new applications. Users are interested in the possibility of usingselectively such a cutting-edge algorithm that may not be run inreal-time nor in lagged-time.

Because of hardware constraints, the time required to automaticallyextract the information of interest from the set of images in the buffercould not be completed in the time that separates two frames coming fromthe acquisition device. In an embodiment, a computationally intensivealgorithm may use a frozen set of image to produce a new enhanced imageor a new enhanced set of images and does it in an iterative manner.

FIG. 2 illustrates several steps of a method implementing video sequencemosaicing incremental algorithm according to an embodiment of thepresent disclosure. Vercauteren et al. showed potential benefits ofusing dedicated video mosaicing techniques to widen the field of view byaligning and fusing many consecutive images from a video sequence, forexample in the context of endomicroscopy. This mosaicing algorithm maynot be run in real-time and works by iteratively refining a mosaicimage. It can thus clearly benefit from the present invention. Infurther detail, still referring to FIG. 2, upon activation of freezemode, the images loaded into the buffer are frozen (step 200), meaningthat the loading of images into the buffer is stopped. Then, the loadedimages (also referred to as frozen images) may first go through aninitialization and preprocessing step 210. This step might for exampleconsist of automatically choosing a subset of the images in the FIFObuffer so that the remainder of the mosaicing algorithm may assume thatall consecutive frames in the subset are overlapping. This may be doneby performing a fast but rough initial registration. A threshold on aquantitative evaluation of the quality of the rough registration can beused to define the subset of overlapping images. Afterwards thefollowing steps may be performed in an iterative manner. Registrationresults are refined (step 220). A freeze test is then performed (step230) in order to determine if the system is still in freeze mode. If thesystem has been switched back to the default not frozen mode (arrow N),registration results might be stored and the processing is halted (step232). Otherwise, a mosaic image is constructed (step 240) and displayed(step 241). A freeze test is then performed (step 250). If the systemhas been switched back to the default not frozen mode (arrow N), thereconstructed mosaic might be stored (step 242) and the processing ishalted. Otherwise, a new refinement step is performed and the process isperformed in an iterative manner.

FIG. 3 is a display illustrating successive results of a video sequencemosaicing incremental algorithm according to an embodiment of thepresent invention. It highlights incremental improvement of an imagemosaic as computed, during a freeze time period. The mosaicing algorithmmay be run on a plurality of frames (for example 26 frames) of a healthyhuman colon acquired in vivo by means of endomicroscopy. Initialalignment may be rather rough and the image mosaic may be a simple imageoverlay (image 300). Then a globally consistent alignment may computed(image 310) and a state-of-the-art image fusion technique may used. Thismay be followed by a mosaic that takes into account motion distortionthat alters endomicroscopy (image 320). Finally a mosaic compensatingfor non-rigid deformations due to interactions between the imaged softtissue and an optical probe of an endomicroscope may be constructed(image 330).

FIG. 4 is a diagram representing schematically steps of a methodimplementing a super-resolution incremental algorithm according to anembodiment of the present disclosure. Patent Application US20070273930showed potential benefit of creating a high resolution image from a setof shifted images, for example in the context of endomicroscopy. Besidesa mechanical device presented there to shift images, super-resolutionmight also be done from uncontrolled motion images. As presented byIrani and Peleg, typical super-resolution algorithms are iterative innature and require a large amount of processing power. In furtherdetail, still referring to FIG. 4, upon activation of the freeze mode,the images loaded into the buffer are frozen (step 400). The freezed setof images may be then registered onto a given reference (step 410). Thealignment might be imposed by the mechanical constraints as in US PatentApplication US20070273930 or might be the results of some imageregistration algorithm. From this alignment a high-resolution image isconstructed in step 420, and displayed in step 421. A freeze test isthen performed in step 430. If the system has been switched back to thedefault not frozen mode (arrow N), the reconstructed high-resolutionimage might be stored (step 422) and the processing is halted. Otherwise(arrow Y), low-resolution images are simulated from the currenthigh-resolution image and knowledge of the imaging system in step 440.The error between the simulated low-resolution images and the actualoriginal low-resolution images is used to improve the currenthigh-resolution image by a back-projection technique, going back to thestep of constructing an high resolution image.

FIG. 5 is a display illustrating successive results of asuper-resolution incremental algorithm according to an embodiment of thepresent disclosure. An image from a frozen set of images is chosen andupsampled to provide an approximation of an high-resolution image (image500). A first and second successive results of iterative improvementsare shown (respectively images 510 and 520).

FIG. 6 is a diagram representing schematically steps of a methodimplementing blood velocity measurement incremental algorithm accordingto an embodiment of the present disclosure. US Patent ApplicationUS20080045848 showed potential benefits of measuring blood velocity froma set of images, for example in the context of endomicroscopy. Aspresented by Perchant et al., blood velocity computation might be doneby a pipeline of processing algorithms that work on a set of consecutiveimages. The complete processing may require a large amount of processingpower. Even though the pipeline is not strictly speaking iterative, itis still incremental. Results of each subcomponent of this pipeline canbe of interest to the user. In further detail, still referring to FIG.6, upon activation of the freeze mode, the images loaded into the bufferare frozen (step 600). A region of interest within one given image maybe automatically tracked and stabilized across the set of frozen images(step 610) resulting in a set of stabilized images. The initial regionof interest might be defined by the user, automatically selected byanother processing algorithm such as a salient region detector, or mightconsist of the complete image. Stabilization results might be stored(step 612) and/or displayed (step 611). A freeze test may be performed(step 620). If the system has been switched back to the default notfrozen mode (arrow N), processing is simply halted. Otherwise (arrow Y),a mean image is computed from the stabilized region of interest sequenceto improve the signal to noise ratio and a vessel segmentation algorithmis performed on the mean stabilized image (step 630). Segmentationresults might be displayed (step 631) and stored (step 632). A freezetest may be performed (step 640). If the system has been switched backto the default not frozen mode (arrow N), processing is simply halted.Otherwise (arrow Y), segmentation is propagated to all images in the setof stabilized images (step 650). Segmentation propagation might bedisplayed (step 651) and/or stored (step 652). A freeze test may beperformed (step 660). If the system has been switched back to thedefault not frozen mode (arrow N), processing is simply halted.Otherwise (arrow Y), blood velocity is computed within the detectedvessels by a dedicated processing algorithm such as a medial linecorrelation method (step 670). Finally the estimated blood velocity isdisplayed (step 671) and/or stored (step 672).

FIG. 7 is a display illustrating successive results of a blood velocitymeasurement incremental algorithm. It highlights progression throughblood velocity measurement processing pipeline as computed by thepreviously described possible embodiment. A given region of interest istracked and stabilized through a sequence in a frozen buffer (images710, 720 and 730). Then, a stabilized mean region of interest image isshown and used to segment the vessel structure present in the region ofinterest (images 740 and 750). In the following step, the segmentationis propagated to the stabilized region of interest sequence (images 760,770 and 780). Finally a graph representing the estimation of bloodvelocity though the freezed sequence as a function of time is displayed(image 790).

In a typical clinical use of endomicroscopy according to the prior art,endoscopic and endomicroscopic images are displayed to a user onseparated displays. Generally, the microscopic imaging probe is visibleon the macroscopic endoscopic view. It may be of clinical interest tofuse the two sources of information and show the microscopic imageswithin their macroscopic context. However, the image processing to fusethe flow of macroscopic and microscopic images cannot be run in realtime.

According to an embodiment of the present disclosure, it may be possibleto fuse information from several acquisition devices. FIG. 8 illustratesseveral steps of a method used to fuse images. For example a first flowof images may be acquired on a first acquisition device and a secondflow of images may be acquired on a second acquisition device. The firstand second acquisition devices may be mechanically coupled so as toacquire images of the same object under observation. In an embodiment,the first and second acquisition devices may be an endo scope and anendomicroscope inserted in an accessory channel of the endoscope so asto acquire simultaneously microscopic and macroscopic images.

More precisely and still referring to FIG. 8, in an embodiment imagesacquired by the endomicroscope (first acquisition device) may be loadedin a first buffer (step 802) while images acquired by the endoscope(second acquisition device) may be loaded in a second buffer (step 803).

During the acquisition, images from the first and second acquisitiondevices may be displayed. The user may select, during the ongoingacquisition, one or more interesting images of the second flow of images(macroscopic images from the endoscope) associated with one or moreimages of the first flow of images (microscopic images from theendomicroscope). The associated images of the first flow of images maytemporally correspond to the selected images of the second flow ofimages. The selection may be carried out for example by clicking on abutton (step 801). The system may store timings, called interestsignals, enabling to retrieve the selected images from the buffer.Alternatively, interesting images among the first and second set ofimages may be selected automatically by an algorithm among the imagesstored in the first and second buffers. For example, one image out often may be automatically selected in the first and second buffers.

In another embodiment, when the freeze command is activated and thefirst and second buffer are frozen, the user may also select imagesamong the first or second sets of images loaded in the first and secondbuffers. For example, the user may review the sets of images loaded inthe first and/or second buffers by displaying said images on a displayunit. For example, an image from the first or second sets of imagesloaded in the frozen buffers may be selected when the image is displayedfor more than a predetermined amount of time.

As described in the previous embodiments, the user may decide that animage processing should be run. Therefore, the user may for examplepress a freeze button, triggering the freeze mode. Entering the freezemode may stop the loading of images in the first and second buffers.When the image selection step is completed and the freeze command isactivated (step 804), the system may perform a detection step (step 805)on one of the selected image. The detection may comprise detecting theendomicroscopic probe on one selected image of the second set of images(i.e. macroscopic images) to obtain a macroscopic processed image. Thedetection result may be displayed (step 806). A freeze test may then beperformed (step 807). If the system is not in freeze mode, the detectionresults may be stored (step 808). If the system is in freeze mode, thesystem proceeds and fuses the image of the first set of image(microscopic image) temporally corresponding to the macroscopic selectedprocessed image (step 809). The fused result may be displayed (step810). The microscopic image may be positioned next to the position atwhich the endomicroscopic probe has been detected. Alternatively,advanced texture mapping technique may be used. A freeze test mayperformed (step 811) and the system may either store the fusion resultand bails out (812) or proceeds according to the above mentioned processwith another selected image.

In an embodiment, a plurality of microscopic images may be fused on amacroscopic image (step 905). This may be performed by propagatinginformation resulting from one or more fusions between macroscopic andmicroscopic corresponding images onto a main macroscopic image.Endoscopic images have a large field of view compared to endomicroscopicimages. Therefore, several microscopic images may potentially be fusedon a macroscopic image. Fusing a supplementary microscopic image on amacroscopic image may preliminary require that the supplementarymicroscopic image is fused to a corresponding second macroscopic imageaccording to the previously described scheme.

FIG. 9 is a display illustrating successive results of a fusionalgorithm according to an embodiment of the present disclosure. Anendoscopic image of interest is selected for processing (step 902), theendomiscroscopic probe is detected and the tip of the probe is displayed(step 903). A fusion according to the previously described scheme isperformed to show the microscopic image associated to the endoscopicimage (step 901) in the macroscopic context (step 904). Furtherprocessing steps are then performed and step 905 illustrates the resultof fusing several endomicroscopic images of interest on a macroscopicimage.

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein. Forexample, the images referred to in the description may be multi spectralimages acquired on a plurality of collection channels of an acquisitiondevice. Accordingly, the scope of the invention should be limited onlyby the attached claims.

1. A method for online processing of images acquired, comprising:acquiring images through a medical device in real-time; displaying theimages in real-time; loading the images into a buffer obtaining a set ofimages; stopping the loading of the images into the buffer based on anevaluation of the real-time displayed images; processing the loaded setof images using an incremental algorithm, wherein the incrementalalgorithm is composed of different subroutines that need to be run oneafter another and which provide intermediate results; displayingsuccessively the intermediate results obtained by processing the set ofimages; and resuming the loading, and stopping the processing, based onan evaluation of said intermediate results, wherein acquiring anddisplaying the images in real-time is continued upon stopping theloading of images into the buffer.
 2. (canceled)
 3. The method accordingto claim 1, wherein the acquired images are derived from a videosequence and wherein the processing comprises iteratively aligning andfusing consecutive images in order to widen the field of view.
 4. Themethod according to claim 1, wherein the acquired images are shiftedimages and the processing comprises iteratively registering said imagesin order to obtain a high resolution image.
 5. The method according toclaim 1, wherein the acquired images are derived from a video sequencerepresenting blood vessels and the processing computes blood velocity.6. The method according to claim 1, wherein the loading is stoppedautomatically.
 7. The method according to claim 6, wherein the loadingis stopped based on a motion detector algorithm.
 8. The method accordingto claim 1, wherein the evaluation of the intermediate results isoperated by a user or automatically.
 9. (canceled)
 10. (canceled) 11.The method according to claim 1, wherein the buffer is a FIFO buffer.12. The method according to claim 1, wherein the intermediate resultsare stored on a storage device.
 13. The method according to claim 1,wherein the intermediate results of the processing are merged to thereal-time displayed images.
 14. The method according to claim 1, whereinthe medical device is a fiber confocal microscope.
 15. An imaging systemcomprising: a medical device for acquiring images in real-time; adisplay device for displaying the images in real-time; a storage devicecomprising a buffer for loading said images; a freeze command to stopthe loading of the images into the buffer upon activation; and aprocessor for processing images, wherein, upon activation of the freezecommand: the processor processes loaded images using an incrementalalgorithm, wherein the incremental algorithm is composed of differentsubroutines that need to be run one after the other and which provideintermediate results, the display device displays successively theintermediate results of said algorithm, the medical device and displaydevice continue to acquire and display in real time the images, and theloading of images into the buffer is resumed, upon deactivation of thefreeze command, based on an evaluation of said intermediate results. 16.The system according to claim 15, wherein the stopping of the loadinginto the buffer is based on an evaluation of said real-time displayedimages.
 17. The system according to claim 15, wherein the medical deviceis a fiber confocal microscope.
 18. The method according to claim 1,wherein the medical device comprises a first acquisition device and asecond acquisition device and the buffer comprises a first buffer and asecond buffer for respectively loading the images acquired on the firstand second acquisition devices and wherein the processing comprisesfusing one or more images loaded in the first buffer on an image loadedin the second buffer.